The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents' Performance in Pediatric Emergency Medicine.

PURPOSE: Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs). METHOD: EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified. RESULTS: 274 residents and 1,891 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations ( r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay. CONCLUSIONS: For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.

Licensing exams in Canada: a closer look at the validity of the MCCQE Part II.

The Medical Council of Canada Qualifying Exam (MCCQE) Part II aims to protect societal interests through examining recently graduated physicians using clinical scenarios with standardized patients. This position paper debates the role of the MCCQE Part II in the national licensing of physicians in Canada by focusing on the consequential validity evidence of this exam and considering future directions through discussing contemporary developments in high stakes examinations. Specifically, this paper compares both MCCQE Part I and Part II in their ability to predict future practice patterns of physicians and generalizability across specialties. In weighing up the evidence this paper considers commonly used counterarguments as well as the financial implications of this exam for both the candidates and the MCC. Finally, it concludes by providing recommendations for future licensing of physicians in Canada. The available consequential validity evidence for MCCQE Part II is limited. Though still limited, MCCQE Part I has more robust evidence that it is a better predictor of future practice patterns compared to with Part II. Combined with a lack of evidence that national licensing examinations lead to graduation of substandard doctors or an improvement of care, and the shift away from assessment of learning towards assessment for learning, the maximum impact of the MCC on safeguarding public's interests will lie in working closely with residency programs and specialty colleges to facilitate a robust assessment program of essential competencies and clinical skills during residency training and specialty certification.

L'examen d'aptitude du Conseil médical du Canada (EACMC), partie II, vise à protéger le public en soumettant les nouveaux diplômés en médecine à une évaluation par le biais de scénarios cliniques avec des patients standardisés. Cet article porte un regard critique quant au rôle de l'EACMC, partie II, dans l'obtention du permis d'exercice en médecine compte tenu des preuves de la validité de l'examen du point de vue de ses conséquences sociales et proposee des orientations futures au regard des développements contemporains en matière d'examens à enjeux élevés. Plus spécifiquement, cet article compare la partie I et la partie II de l'EACMC quant à leur capacité à prédire les tendances dans la pratique future des médecins et à leur caractère généralisable aux diverses spécialités. En soupesant les preuves, l'auteure examine les contre-arguments couramment évoqués ainsi que les implications financières de l'examen pour les candidats et pour le CMC. En conclusion, notre travail formule des recommandations pour l'octroi futur de titres menant à l'obtention du permis d'exercer aux médecins au Canada. Les preuves quant à la validité des conséquences sociales de l'EACMC, partie II sont limitées. Bien qu'également limitées, les preuves de la validité de l'EACMC, partie I, sont plus solides que celles de l'EACMC, partie II, en ce qui a trait à sa valeur prédictive des tendances futures dans la pratique des médecins. En l'absence de données probantes indiquant que les examens mènent soit à l'octroi d'un titre menant à l'obtention d'un permis de pratique en médecine qui ne satisfont pas aux exigences soit à l'amélioration des soins, et compte tenu de l'abandon de l'évaluation de l'apprentissage au profit de l'évaluation pour l'apprentissage, la meilleure manière pour le CMC de favoriser la protection du public serait de travailler en étroite collaboration avec les programmes de résidence et les collèges de spécialité afin de faciliter la mise en place d'un solide programme d'évaluation des compétences essentielles et des compétences cliniques pendant la formation en résidence et à l'occasion de formation continue spécialisée.

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review.

PURPOSE: To collect and examine-using an argument-based validity approach-validity evidence of questionnaire-based tools used to assess physicians' clinical, teaching, and research performance. METHOD: In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians' professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians' clinical, teaching, and research performance. Using Kane's validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications. RESULTS: They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane's validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing. CONCLUSIONS: Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians' clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

Defining and Adopting Clinical Performance Measures in Graduate Medical Education: Where Are We Now and Where Are We Going?

Assessment and evaluation of trainees' clinical performance measures is needed to ensure safe, high-quality patient care. These measures also aid in the development of reflective, high-performing clinicians and hold graduate medical education (GME) accountable to the public. Although clinical performance measures hold great potential, challenges of defining, extracting, and measuring clinical performance in this way hinder their use for educational and quality improvement purposes. This article provides a way forward by identifying and articulating how clinical performance measures can be used to enhance GME by linking educational objectives with relevant clinical outcomes. The authors explore four key challenges: defining as well as measuring clinical performance measures, using electronic health record and clinical registry data to capture clinical performance, and bridging silos of medical education and health care quality improvement. The authors also propose solutions to showcase the value of clinical performance measures and conclude with a research and implementation agenda. Developing a common taxonomy of uniform specialty-specific clinical performance measures, linking these measures to large-scale GME databases, and applying both quantitative and qualitative methods to create a rich understanding of how GME affects quality of care and patient outcomes is important, the authors argue. The focus of this article is primarily GME, yet similar challenges and solutions will be applicable to other areas of medical and health professions education as well.

The Association Between Residency Learning Climate and Inpatient Care Experience in Clinical Teaching Departments in the Netherlands.

PURPOSE: To examine the association between residency learning climate and inpatient care experience. METHOD: The authors analyzed 1,201 evaluations of the residency learning climate (using the Dutch Residency Educational Climate Test questionnaire) and 6,689 evaluations of inpatient care experience (using the Consumer Quality Index Inpatient Hospital Care questionnaire) from 86 departments across 15 specialties in 18 hospitals in the Netherlands between 2013 and 2014. The authors used linear hierarchical panel analyses to study the associations between departments' overall and subscale learning climate scores and inpatient care experience global ratings and subscale scores, controlling for respondent- and department-level characteristics and correcting for multiple testing. RESULTS: Overall learning climate was not associated with global department ratings (b = 0.03; 95% confidence interval -0.17 to 0.23) but was positively associated with specific inpatient care experience domains, including communication with doctors (b = 0.11; 0.02 to 0.20) and feeling of safety (b = 0.09; 0.01 to 0.17). Coaching and assessment was positively associated with communication with doctors (b = 0.22; 0.08 to 0.37) and explanation of treatment (b = 0.22; 0.08 to 0.36). Formal education was negatively associated with pain management (b = -0.16; -0.26 to -0.05), while peer collaboration was positively associated with pain management (b = 0.14; 0.03 to 0.24). CONCLUSIONS: Optimizing the clinical learning environment is an important step toward ensuring high-quality residency training and patient care. These findings could help clinical teaching departments address those aspects of the learning environment that directly affect patient care.

The Association Between Learning Climate and Adverse Obstetrical Outcomes in 16 Nontertiary Obstetrics-Gynecology Departments in the Netherlands.

PURPOSE: To investigate the association between learning climate and adverse perinatal and maternal outcomes in obstetrics-gynecology departments. METHOD: The authors analyzed 23,629 births and 103 learning climate evaluations from 16 nontertiary obstetrics-gynecology departments in the Netherlands in 2013. Multilevel logistic regressions were used to calculate the odds of adverse perinatal and maternal outcomes, by learning climate score tertile, adjusting for maternal and department characteristics. Adverse perinatal outcomes included fetal or early neonatal mortality, five-minute Apgar score < 7, or neonatal intensive care unit admission for ≥ 24 hours. Adverse maternal outcomes included postpartum hemorrhage and/or transfusion, death, uterine rupture, or third- or fourth-degree perineal laceration. Bias analyses were conducted to quantify the sensitivity of the results to uncontrolled confounding and selection bias. RESULTS: Learning climate scores were significantly associated with increased odds of adverse perinatal outcomes (aOR 2.06, 95% CI 1.14-3.72). Compared with the lowest tertile, departments in the middle tertile had 46% greater odds of adverse perinatal outcomes (aOR 1.46, 95% CI 1.09-1.94); departments in the highest tertile had 69% greater odds (aOR 1.69, 95% CI 1.24-2.30). Learning climate was not associated with adverse maternal outcomes (middle vs. lowest tertile: OR 1.04, 95% CI 0.93-1.16; highest vs. lowest tertile: OR 0.98, 95% CI 0.88-1.10). CONCLUSIONS: Learning climate was associated with significantly increased odds of adverse perinatal, but not maternal, outcomes. Research in similar clinical contexts is needed to replicate these findings and explore potential mechanisms behind these associations.

Closing the patient experience chasm: A two-level validation of the Consumer Quality Index Inpatient Hospital Care.

BACKGROUND: Evaluation of patients' health care experiences is central to measuring patient-centred care. However, different instruments tend to be used at the hospital or departmental level but rarely both, leading to a lack of standardization of patient experience measures. OBJECTIVE: To validate the Consumer Quality Index (CQI) Inpatient Hospital Care for use on both department and hospital levels. DESIGN: Using cross-sectional observational data, we investigated the internal validity of the questionnaire using confirmatory factor analyses (CFA), and the generalizability of the questionnaire for use at the department and hospital levels using generalizability theory. SETTING AND PARTICIPANTS: 22924 adults hospitalized for ≥24 hours between 1 January 2013 and 31 December 2014 in 23 Dutch hospitals (515 department evaluations). MAIN VARIABLE: CQI Inpatient Hospital Care questionnaire. RESULTS: CFA results showed a good fit on individual level (CFI=0.96, TLI=0.95, RMSEA=0.04), which was comparable between specialties. When scores were aggregated to the department level, the fit was less desirable (CFI=0.83, TLI=0.81, RMSEA=0.06), and there was a significant overlap between communication with doctors and explanation of treatment subscales. Departments and hospitals explained ≤5% of total variance in subscale scores. In total, 4-8 departments and 50 respondents per department are needed to reliably evaluate subscales rated on a 4-point scale, and 10 departments with 100-150 respondents per department for binary subscales. DISCUSSION AND CONCLUSIONS: The CQI Inpatient Hospital Care is a valid and reliable questionnaire to evaluate inpatient experiences in Dutch hospitals provided sufficient sampling is done. Results can facilitate meaningful comparisons and guide quality improvement activities in individual departments and hospitals.

Validation of the INCEPT: A Multisource Feedback Tool for Capturing Different Perspectives on Physicians' Professional Performance.

INTRODUCTION: Multisource feedback (MSF) instruments are used to and must feasibly provide reliable and valid data on physicians' performance from multiple perspectives. The "INviting Co-workers to Evaluate Physicians Tool" (INCEPT) is a multisource feedback instrument used to evaluate physicians' professional performance as perceived by peers, residents, and coworkers. In this study, we report on the validity, reliability, and feasibility of the INCEPT. METHODS: The performance of 218 physicians was assessed by 597 peers, 344 residents, and 822 coworkers. Using explorative and confirmatory factor analyses, multilevel regression analyses between narrative and numerical feedback, item-total correlations, interscale correlations, Cronbach's α and generalizability analyses, the psychometric qualities, and feasibility of the INCEPT were investigated. RESULTS: For all respondent groups, three factors were identified, although constructed slightly different: "professional attitude," "patient-centeredness," and "organization and (self)-management." Internal consistency was high for all constructs (Cronbach's α ≥ 0.84 and item-total correlations ≥ 0.52). Confirmatory factor analyses indicated acceptable to good fit. Further validity evidence was given by the associations between narrative and numerical feedback. For reliable total INCEPT scores, three peer, two resident and three coworker evaluations were needed; for subscale scores, evaluations of three peers, three residents and three to four coworkers were sufficient. DISCUSSION: The INCEPT instrument provides physicians performance feedback in a valid and reliable way. The number of evaluations to establish reliable scores is achievable in a regular clinical department. When interpreting feedback, physicians should consider that respondent groups' perceptions differ as indicated by the different item clustering per performance factor.

Revisiting the D-RECT tool: Validation of an instrument measuring residents' learning climate perceptions.

INTRODUCTION: Credible evaluation of the learning climate requires valid and reliable instruments in order to inform quality improvement activities. Since its initial validation the Dutch Residency Educational Climate Test (D-RECT) has been increasingly used to evaluate the learning climate, yet it has not been tested in its final form and on the actual level of use - the department. AIM: Our aim was to re-investigate the internal validity and reliability of the D-RECT at the resident and department levels. METHODS: D-RECT evaluations collected during 2012-2013 were included. Internal validity was assessed using exploratory and confirmatory factor analyses. Reliability was assessed using generalizability theory. RESULTS: In total, 2306 evaluations and 291 departments were included. Exploratory factor analysis showed a 9-factor structure containing 35 items: teamwork, role of specialty tutor, coaching and assessment, formal education, resident peer collaboration, work is adapted to residents' competence, patient sign-out, educational atmosphere, and accessibility of supervisors. Confirmatory factor analysis indicated acceptable to good fit. Three resident evaluations were needed to assess the overall learning climate reliably and eight residents to assess the subscales. CONCLUSION: This study reaffirms the reliability and internal validity of the D-RECT in measuring residency training learning climate. Ongoing evaluation of the instrument remains important.

Still dizzy after all these years: a 90-year-old woman with a 54-year history of dizziness.

This report presents the case of a 90-year-old female with a 54-year history of dizziness, which has been exhaustively investigated. Over the years, the patient made 59 visits to her family doctor and 18 visits to various specialists, as well as emergency department visits and hospitalisations. In detailing the exhaustive investigations and referrals that the patient has undergone over many years (with inconclusive results), this case illustrates the myriad challenges in diagnosing and treating chronic dizziness in the older. The authors suggest that, in complex cases characterised by multimorbidity and polypharmacy, a function-oriented approach is indicated. In place of the conventional 'diagnose and treat' model, a functional approach to ongoing care emphasises the symptom management, improvement of function and quality of life. To optimise patient outcomes, an interprofessional team approach is preferred.